System and apparatus for haptically enabled three-dimensional scanning

ABSTRACT

Described herein is a three-dimensional scanning system that features a camera integrated with a user-guided haptic interface device. The system allows an operator, through the sense of touch, to intuitively and interactively identify optimum locations for obtaining images or scans of an object. The system then assembles these scans to produce a virtual three-dimensional representation of the object with a high degree of accuracy and with a minimum of data artifacts.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims the benefit of U.S. Provisional PatentApplication No. 61/392,418, filed on Oct. 12, 2010, and U.S. ProvisionalPatent Application No. 61/426,729, filed on Dec. 23, 2010, thedisclosures of which are hereby incorporated by reference herein intheir entireties.

FIELD OF THE INVENTION

This invention relates generally to scanning systems for constructingthree-dimensional models of objects. More particularly, in certainembodiments, the invention relates to a scanning system that includes ascanning module integrated within a user-guided haptic interface device.

BACKGROUND OF THE INVENTION

A common requirement for Dental CAD/CAM systems is the acquisition of atrue three-dimensional representation of the patient situation—that is,the shape of the patient's existing teeth, gums and palette—with theappropriate degree of accuracy for the prosthetic that is to bedesigned. Three-dimensional scanning systems sold to dental labs mostoften digitize the “stone” model, made by plaster casting the patientimpression made by the dentist. In some cases, the patient impression isdigitized directly. Three-dimensional scanning systems sold directly todentists often employ “intra-oral scanning” techniques where thedigitizer is inserted directly into the patient's mouth.

Different range sensing techniques have been used to engineer dental labscanners and intra-oral scanners, including: triangulation, phase-shiftreconstruction, conoscopic holography, confocal microscopy, and time offlight. Most commercial dental lab scanners use triangulation andphase-shift reconstruction. These both work through projectingstructured light onto the object to be digitized, capturing an image (orimages) with a frame grabber, and then reconstructing the image on thecomputer to produce Point Cloud (X, Y, Z) data relative to the point ofview of the image capture device(s).

Conoscopic holography is based on crystal optics and interferencepatterns generated by interacting polarized light rays. The NOBELBIOCARE™ dental lab scanner uses this technique.

Confocal microscopy uses an image capture system with a very narrowfield of focus and then varies the focal plane in a known sequence.Several intra-oral scanners, such as the ITERO™ by CADENT™, use theconfocal principle to construct Point Cloud data.

Time of flight systems direct light (usually from a laser source)against an object and measure the amount of time to detect thereflection. Since the speed of light c is constant, the distance to theobject may be calculated. Because the precision of the time measurementis limited, and the accuracy requirements for dental scanners isrelatively high (in the range of 10-30 microns), time of flight has notyet been used for dental scanners.

Three-dimensional scanning is largely done with hardware and softwarethat is dedicated to scanning, rather than with general purpose hardwareand software. Three-dimensional scanning devices are generallyclassified according to the underlying technology, such as white light,non-white light, point, line, and phase change. Three-dimensionalscanning devices may operate in one of three modes: fully automatic,semi-automatic, and manual.

One of the central problems in capturing scan data for analysis increating a three-dimensional model is that of controlling cameraposition and orientation relative to the object being scanned. Given afixed camera focal length based on the lens configuration, if the camerais at the optimal focal distance from the object, then mathematicallyprecise scan data may be extracted from the images. By combining asufficient number of such images, and by knowing the position of theobject and the camera locations and orientations from which the imageswere collected, a high quality three-dimensional reconstruction may becreated. Failing to control camera position and orientation, however,may lead to ambiguity in the collected data, thereby rendering thethree-dimensional reconstruction an approximation with unknown accuracy.

There is a need for improved methods, systems, and apparatus forscanning an object to produce a virtual three-dimensional representationof the object.

SUMMARY OF THE INVENTION

Described herein is a three-dimensional scanning system that features acamera integrated with a user-guided haptic interface device. The systemallows an operator, through the sense of touch, to intuitively andinteractively identify optimum locations for obtaining images or scansof an object. The system then assembles these scans to produce a virtualthree-dimensional representation of the object with a high degree ofaccuracy and with a minimum of data artifacts. In the dental field, forexample, the object being scanned may be the interior of a patient'smouth (or an impression or cast thereof, such as a dental stone), andthe haptic interface device may include a stylus with a camera at theend. The three-dimensional representation of the scanned object may beused, for example, for preparation of dentures, crowns, dentalappliances, implants, or other dental devices, custom fitted for thepatient.

One or more haptic guides facilitates acquisition of useful images ofthe object by the user. For example, the movement of a haptic interfacedevice being manipulated by a user about the object being scanned may beconstrained to (and/or may “snap to”) a particular 2D or 3D surface,region, line, point, and/or orientation in space in relation to theobject being scanned, in order to guide the user to obtain useful datafor construction of the virtual 3D representation of the object. In theexample given above, the haptic guide(s) would constrain (eitherstrongly or weakly) the movement of the stylus to camera locations andorientations from which useful images may be obtained. The user mayacquire images at such locations, for example, by pressing a button onthe haptic interface device. As the user obtains more images, the hapticguide(s) may be updated based on the newly acquired images. By acquiringthree-dimensional data in real-time, the haptically guided scanningsystem allows the operator to easily identify and interactively fill invoids of the constructed virtual representation (model). By watching themodel being filled-in on a display monitor in real-time, and by sensingin real-time the haptic guides that direct the user to optimum dataacquisition locations, the operator feels as if he is “crayoning” theobject being scanned to make the three-dimensional details emerge on thescreen.

This intuitive, manually guided scanning system is faster than automatedscanning processes and provides better, more accurate resolution ofsurfaces, particularly for objects having high curvature, undercuts,and/or deep or discontinuous regions, such as gaps between teeth and/orimpressions, for example. Because of the crayoning effect experienced bythe user, the user may interactively scan until the model is completelyfilled-in, so that there is little or no need for post-acquisition holefilling or other artifact removal. In certain embodiments, the systemmay be used to scan regions (e.g., interior regions of the body) forwhich adequately detailed impressions cannot be readily obtained. Forexample, three-dimensional representations of thusly scanned regions maybe used for production of custom joints, prostheses, and/or othermedical appliances.

Besides the application of scanning dental stones, the technologydescribed herein is generally applicable to other dental, medical, orreverse engineering scanning tasks. In certain embodiments, theinvention provides systems and methods for real-time imagereconstruction, thereby enabling use of the Haptic Scanner to create 3Dpoint clouds for moving or soft tissues.

In the dental field, the Haptic Scanner described herein can be used,for example, for intra-oral scanning, wherein the dentist directlyimages the patient's teeth (and mouth interior) without creating theintermediate impression or stone. In this case, the haptic device mayinclude an extension joint so that the camera and projector scannercombination can be inserted into the mouth and access the distal areaswhere the molar teeth are positioned.

By using a larger haptic device, such as PHANTOM® Premium 3.0,manufactured by SensAble Technologies, Inc. of Wilmington, Mass., it ispossible to use the haptic scanner to acquire 3D data of a person'sface. This is useful in dentistry, for example, to measure “eye tosmile” parameters that are used when a patient needs full dentures; incranial-maxilla facial surgery planning; or as an input for a facialrecognition security application, for example.

A larger haptic device also enables three-dimensional scanning fororthotics and prosthetics applications. For example, the shape of apatient's residual limb can be digitized as part of the process formaking an artificial arm or leg. The haptic device may also be used toscan the shape of the head for cranial helmets, the shape of the footfor custom orthotics, and the overall shape of the torso to createcustom back braces.

The haptic device can also be arranged in a typical “master/slave”configuration where the haptic scanner is attached to a slave devicethat is controlled through a haptic master such as the PHANTOM® DESKTOP™or PHANTOM® OMNI®, manufactured by SensAble Technologies, Inc. Thisconfiguration can be more convenient for the user in many situationswhere the object to be scanned is either very large, very small, orotherwise only remotely accessible to the user; but the fundamental“crayoning” interface should remain intact.

On a commercial scale, scanning for reverse engineering is one of thelargest application areas, and the haptic scanner is well suited to thisapplication as well. Engineering parts or shapes may contain manyconcave or “hidden” features where the flexibility provided by theinvention's haptically guided scanning capability will provide an easyand intuitive user interface. In one embodiment, the scanning system isused to create a three-dimensional virtual model of an existing physicalpart for use in 3D CAD, CAM, CAE or other software. The virtual modelmay be used, for example, to analyze how a product works, how much itcosts, what it consists of, and/or to identify potential patentinfringement.

In one aspect, the invention is directed to a system for hapticallyenabled, three-dimensional scanning of an object, the system comprising:a haptic interface device configured to provide haptic feedback to auser and receive input from the user during movement of an implement(e.g., a stylus) of the haptic interface device during three-dimensionalscanning of an object, wherein the implement comprises a camera; agraphical interface configured to provide graphical feedback to the userduring three-dimensional scanning of the object; and a three-dimensionalscanning application in communication with the haptic interface deviceand the graphical interface, wherein the scanning application isconfigured to: (a) obtain an image of the object upon activation of auser command (e.g., upon the pressing of a button of the hapticinterface device by the user); (b) determine a haptic guide at one ormore positions and/or orientations in space in relation to the objectfrom which advantageous acquisition of images is possible forconstruction of a three-dimensional virtual representation of theobject, wherein the haptic guide is determined using at least the imageobtained in step (a); (c) deliver force to the user via the hapticinterface device according to the haptic guide (e.g., constrain themovement of the implement by the user to the advantageous positionsand/or orientations corresponding to the haptic guide); (d) repeat oneor more of steps (a) to (c) as additional images of the object areacquired; and (e) produce a three-dimensional virtual representation ofthe object using at least a subset of the images obtained. In certainembodiments, the three dimensional scanning application comprises amemory that stores code defining a set of instructions and a processorthat executes the instructions. In one embodiment, the system isconfigured for use in a minimally invasive surgery (MIS) system.

In another aspect, the invention is directed to a method for hapticallyenabled, three-dimensional scanning of an object, the method comprising:(a) obtaining an image of the object upon activation of a user command(e.g., upon the pressing of a button of the haptic interface device bythe user), wherein the image is obtained during manipulation by the userof an implement (e.g., stylus) of a haptic interface device about theobject; (b) delivering graphical feedback to the user via a graphicaldisplay during the manipulation of the implement of the haptic interfacedevice by the user; (c) determining a haptic guide at one or morepositions and/or orientations in space in relation to the object fromwhich advantageous acquisition of images is possible for construction ofa three-dimensional virtual representation of the object, wherein thehaptic guide is determined using at least the image obtained in step(a); (d) delivering a force to the user via the haptic interface deviceaccording to the haptic guide (e.g., constraining the movement of theimplement by the user to the advantageous positions and/or orientationscorresponding to the haptic guide); (e) repeating one or more of steps(a) to (d) as additional images of the object are acquired; and (f)producing a three-dimensional virtual representation of the object usingat least a subset of the images obtained. The description of elements ofthe embodiments above can be applied to this aspect of the invention aswell.

In certain embodiments, the object is a tooth, a human face, a residuallimb, an anatomical structure (e.g., an organ), or a mechanical part.The method may also include analyzing the three-dimensional virtualrepresentation to reverse engineer the object.

In yet another aspect, the invention relates to an apparatus forscanning an object. The apparatus includes a user connection element, ascanning module associated with the user connection element, anactuator, a linkage physically linking the user connection element tothe actuator, and a processor for determining force delivered to theuser connection element by the actuator to restrict or guide movement ofthe user connection element according to a haptic guide. The scanningmodule acquires a plurality of images of the object during the scan. Thedescription of elements of the embodiments above can be applied to thisaspect of the invention as well.

In certain embodiments, the user connection element includes anextension joint for scanning a difficult to reach object, which mayinclude a tooth, a human face, a residual limb, an anatomical structure,and/or a mechanical part. The haptic guide may be configured tofacilitate movement of the user connection element by the user to one ormore positions and/or orientations in space in relation to the objectfrom which advantageous acquisition of images is possible forconstruction of a three-dimensional virtual representation of the objectfrom the acquired images. The haptic guide may be updated according toone or more previously obtained images of the object. In certainembodiments, the scanning module is configured for use in a minimallyinvasive surgery (MIS) system.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects and features of the invention can be better understood withreference to the drawings described below, and the claims. The drawingsare not necessarily to scale, emphasis instead generally being placedupon illustrating the principles of the invention. In the drawings, likenumerals are used to indicate like parts throughout the various views.

While the invention is particularly shown and described herein withreference to specific examples and specific embodiments, it should beunderstood by those skilled in the art that various changes in form anddetail may be made therein without departing from the spirit and scopeof the invention.

FIG. 1 is a schematic perspective view of a haptically guided scanningsystem, according to an illustrative embodiment of the invention.

FIG. 2 is a flowchart depicting a haptically guided scanning system,according to an illustrative embodiment of the invention.

FIG. 3 is a flowchart depicting the data flow for a haptically guidedscanning system, according to an illustrative embodiment of theinvention.

FIG. 4 is a schematic perspective view of a haptically guided scanningsystem that includes an integrated fixture with embedded fiducial pointsthat establish a scanning reference, according to an illustrativeembodiment of the invention.

FIG. 5 is a schematic view of a haptic device, according to anillustrative embodiment of the invention.

FIG. 6 is a schematic view of a haptic device that includes an extensionjoint and an add-on scanning module, according to an illustrativeembodiment of the invention.

DETAILED DESCRIPTION

It is contemplated that devices, apparatus, systems, methods, andprocesses of the claimed invention encompass variations and adaptationsdeveloped using information from the embodiments described herein.Adaptation and/or modification of the devices, apparatus, systems,methods, and processes described herein may be performed by those ofordinary skill in the relevant art.

Throughout the description, where devices and systems are described ashaving, including, or comprising specific components, or where processesand methods are described as having, including, or comprising specificsteps, it is contemplated that, additionally, there are devices andsystems of the present invention that consist essentially of, or consistof, the recited components, and that there are processes and methodsaccording to the present invention that consist essentially of, orconsist of, the recited processing steps.

It should be understood that the order of steps or order for performingcertain actions is immaterial so long as the invention remains operable.Moreover, two or more steps or actions may be conducted simultaneously.

The mention herein of any publication, for example, in the Backgroundsection, is not an admission that the publication serves as prior artwith respect to any of the claims presented herein. The Backgroundsection is presented for purposes of clarity and is not meant as adescription of prior art with respect to any claim.

As discussed above, one of the central problems in capturing scan datafor analysis in creating a three-dimensional model is that ofcontrolling camera position and orientation relative to the object beingscanned. Failing to control camera position and object position may leadto ambiguity in the collected data, thereby rendering thethree-dimensional model an approximation with unknown accuracy.

The methods, systems, and apparatus described herein address theseissues by enabling an operator to control camera position andorientation through the use of haptic guides. A haptic guide may be, forexample, a constraint (e.g., weak or strong) that limits or guides auser's movement of an implement of the haptic interface device, e.g., astylus, about the object being scanned. The haptic guide is “felt” bythe user via force feedback delivered to the user as the user moves theimplement of the haptic interface device. Examples of such constraintsare “hotspots,” gravity wells, and other haptic cues. The haptic guidemay be, for example, a point constraint, line constraint, 2D or 3Dsurface constraint, and/or orientation constraint. Haptic detents aredents or bumps that are sensed by a user moving an implement of thehaptic interface device in space in the vicinity of certain points inspace. The haptic guide may provide a “snap-to” an advantageousposition, line, or surface, for image/scan data acquisition. Such ahaptic guide may have a “snap-distance” associated with the guide,whereby a user will sense, via force feedback delivered by the hapticinterface device, a “force field” encouraging movement of the hapticinterface device implement (e.g., stylus) to the advantageous positionor orientation.

Images obtained from a current camera position are analyzed by any oneof several methods, e.g., via structured light, to determine how thecamera should be repositioned to improve the source data being captured.The result of this analysis is then transformed into haptic guidance toassist the operator in repositioning the camera to an improved location.This cycle of capturing, analyzing, and haptically guiding the camera toan improved location is repeated—for example, at hundreds of cycles persecond—until the camera is guided to a position within a preferredtolerance of optimal location and/or orientation for image capture, anda single frame is captured to serve as one of several in the final setto be analyzed. This process is again repeated with the operatorchoosing a new vantage point to initialize each iteration and beingguided haptically to an additional optimal location at which pointanother final image is collected. The image collection is repeated untila sufficient number of high quality images have been collected to enablea high quality three-dimensional reconstruction.

In one embodiment, to perform a scan, the object to be scanned is placedon a fixture or a simple surface. The operator then navigates a styluswhich is part of a haptic interface device to positions inthree-dimensional space along the haptic guide around the object. Thehaptic guide may act as a “force field” limiting motion of the stylus topositions and/or orientations from which useful optical/image data ofthe object being scanned may be obtained. Each time the operatordetermines that a camera position would be good for capture, images arecaptured and become part of the scan. The operator may then move toadditional locations in space around the object to capture more images.The system may update the haptic guide as more images/data are obtained,modifying the haptic guide in light of the additional data obtained. Theprocess continues until a sufficient number of images have beencollected to enable high quality three-dimensional reconstruction of thethree-dimensional object being scanned. The systems described herein maybe used to scan many different types of objects, both large and small,such as bones, teeth, and tissue, and/or impressions or casts thereof.In addition, the systems may be used to scan objects that reside insideor outside of a patient's body, or wherever an operator may behaptically guided while scanning an object.

Referring to FIGS. 1 and 2, in one embodiment, a haptically guidedscanning system 10 includes a haptic device 12, a three-dimensionalscanning module 14, and a computer with haptic software 16. The hapticdevice 12 may be a three degree of freedom or six degree of freedomhaptic device. In one embodiment, the haptic device 12 is the same as orsimilar to, for example, PHANTOM® OMNI® or DESKTOP™, manufactured bySensAble Technologies, Inc. The haptic device 12 includes a pen orstylus 18 that may be held and manipulated by an operator of the hapticdevice 12. Integrated into a tip or other portion of the stylus 18 isthe three-dimensional scanning module 14. The computer with hapticsoftware 16 receives scan data, such as image data, from the scanningmodule 14 and provides the operator with force feedback by guiding themovement of the stylus 18. Images from the scanning module 14 aredisplayed in a video window of a computer monitor 19 or other displaydevice.

In the embodiment depicted in FIG. 2, the scanning module 14 includes aprojector 20 or laser for projecting structured light, such as linepatterns, and at least one camera 21 for viewing the structured light asit strikes an object. The projector 20 may be a custom high speeddigital light projection (DLP) projector. In one embodiment, theprojector 20 is in the form of an annular ring surrounding the cameralens. The light may include a mask for projecting structured light. Theat least one camera 21 may be a high speed digital camera with anattached fixed focus lens, a high resolution CCD camera, and/or athree-dimensional range camera.

Although the scanning module 14 depicted in FIG. 2 uses triangulationbased on projected line patterns, other scanning methods may beutilized, such as phase-shift reconstruction, conoscopic holography,confocal microscopy, and time of flight. In one embodiment, the scanningmodule 14 is a white light phase change scanner.

Referring to FIG. 2, data from the camera 21 is processed using one ormore integrated circuits, such as field-programmable gate arrays(FPGAs), and transferred to the computer 16 using a 1000 Base T Ethernetconnection 22. A flexible connection 24 is used to transfer data from ahead interface FPGA 26 at a head of the haptic device 14 to a baseinterface FPGA 28 at a base of the haptic device 14. The computer 16 mayinclude an expansion card 30, such as a PCI-E interface.

The computer and haptic software 16 are used to guide a manual scanningprocess in which an operator aims the scanning module 14 at an object 32to be scanned. Specifically, the computer and software 16 guide theoperator during the scan by providing force feedback through the stylus18. This haptic guidance allows the operator to maintain the scanningmodule 14 at or near the proper orientation and optimal distance fromthe object 32. In one embodiment, the haptic device 12 includes anintegrated extension that provides a reference frame for the scanningoperations. The integrated extension may include fiducial marks.

Operation of the scanning system 10 begins during a setup phase when theobject 32 is placed in front of the haptic device 12 (or on theintegrated extension of the device), and the scanning application orsoftware is started. Holding the stylus 18, the operator pulls an arm 34of the haptic device 12 away from the object 32, out to its farthestposition. With the scanning module 14 aimed at the object 32 (or atfiducial marks included on the integrated extension), the operatorstarts the scanning operation by pushing a stylus button. Images fromthe camera 21 are displayed on the computer monitor 19 and providevisual feedback to the operator during the scan. By viewing theseimages, the operator may see digitized images and a structured lightfringe pattern projected from a DLP projector.

After the scanning process has been initiated, the operator moves thestylus 18 towards the object 32 until a haptic guide is felt. The hapticguide helps the user maintain the optimal distance and orientationbetween the camera 21, such as a three-dimensional range camera, and theobject 32. At this point, the monitor 19 displays a small section of thedigitized three-dimensional model and a corresponding haptic guidancesurface.

Once the haptic guide has been contacted, the operator uses visual andforce feedback to move the stylus 18 and scanning module 14 along thehaptic guide to reveal a three-dimensional model or image of the object32. As more points of the three-dimensional model are digitized, thehaptic guide is updated and expanded accordingly. With this approach,the operator is free to focus efforts on digitizing and resolving areasor points of clinical significance. In addition, by avoiding features ofthe object 32 that do not need to be scanned and/or included in thethree-dimensional model (e.g., regions that are not needed for creatinga final prosthetic), subsequent trimming of the model may beunnecessary. Once the object 32 has been scanned as desired, theoperator releases the stylus button to indicate that the scan iscomplete.

During the scanning process, scan data is acquired and analyzed to formPoint Cloud data for the object 32, which may be any object, such as abone, tissue, an organ (e.g., a gall bladder or a prostate gland), atooth, or a dental stone. The Point Cloud data may be used both tointeractively visualize a preliminary scan and to provide hapticguidance during the actual scan. For example, three-dimensional scanningsystems generally have an ideal depth of field and range of focus withinwhich the Point Cloud data will be most accurate. The haptic guidanceallows the operator to enforce the ideal object to scanning moduledistance. Additionally, since the three-dimensional acquisition occursin real time, haptic guides may be programmed to help the stylus 18settle and/or to help the user hold the stylus 18 still during the dataacquisition stage.

Because the haptically guided scanning system 10 is intended to acquirethree-dimensional data in real-time, the operator may easily identifyand fill voids in the captured model interactively. By watching themodel fill-in interactively, the operator feels as if he is “crayoning”the object 32 to make the three-dimensional details emerge.

The scanning system 10 presented herein offers several advantages overother scanning systems used for similar purposes. For example, thesystem described herein provides an intuitive, manually guided scanningprocess that is faster than automated or semi-automated processes. Thesystem also provides better, more accurate resolution of surfaces,particularly for objects 32 having high curvature and/or deep ordiscontinuous regions, such as gaps between teeth and/or impressions.Due to the crayoning aspect described above, the operator mayinteractively scan until the model is completely filled-in, so thatthere is little or no need for post-acquisition hole filling. Inaddition, the cost of the scanning hardware is low because it is basedon a simple, short-focus optical system. The scanning system 10 alsoworks with any computer capable of haptics. Finally, high quality scansmay be obtained using existing three-dimensional scanning technologies,such as triangulation and/or structured light.

Referring to FIG. 2, in another aspect, the real-time performance of theimage capture, display, filtering, and meshing is maximized throughpipelining. An advantage of choosing three-dimensional scannertechnology that is based on digital cameras and embedded projection,such as DLP by Texas Instruments, Inc., is that costs will naturallyfall and the technology will be pushed ahead by the large consumer,digital video, and gaming markets. The systems described herein leveragethese trends while maintaining real-time, three-dimensional performanceby pipelining the data flow through a Graphics Processing Unit (GPU) 36,such as the NVIDIA® line of graphics cards. The NVIDIA® QUADRO® orGEFORCE® boards can be found commonly on graphics oriented workstationsthat are optimized for three-dimensional graphics or gaming.

The GPU architecture is well suited to address problems that may beexpressed as “data-parallel” computations—where the same machineinstructions are executed on many data elements all at once. Thisremoves the typical CPU requirements for sophisticated flow control.Applications that process large data sets such as arrays may benefitfrom a data-parallel programming model. Three-dimensional renderingalgorithms that process large numbers of pixels and vertices may also bemapped to parallel threads. Similarly, image processing algorithms, suchas those used for video post-processing, image scaling, or patternrecognition, may also be accelerated by data-parallel processing.

FIG. 3 depicts the data flow for an embodiment that utilizes white-lighttriangulation. A transformation matrix M is derived (step 40) based onthe x, y, z position and gimbal pitch, roll, and yaw of the stylus 18 ofthe haptic device 12. The scanning module 14 is used to capturestructured light images of the object 32 being scanned. To correct imagequality problems, such as optical distortion, or improper gain and/orcontrast, two dimensional image processing is applied (step 42). An X,Y, Z Point Cloud is constructed (step 44) using triangulation, based onthe captured images, and the transformation matrix M is applied. Next,the Point Cloud is filtered (step 46) based on local, regionalinformation and current overall Point Cloud collection. The Point Cloudis then aligned with existing collection of Point Clouds (e.g., using anIterated Closest Point ICP algorithm) and data is merged (step 48).During the merge step (step 48), a “confidence measure” for the PointCloud points is generated. The model is then updated and displayed (step50) on the computer monitor 19 for the operator, and the process isrepeated until the operator is satisfied that all voids have beenfilled.

The data flow steps described above may not all be necessary and/or thedata flow may include additional steps. For example, Point Cloud datamay be thinned or merged to remove overlapping points. In addition,surfaces may be defined through triangulation or meshing of the PointCloud collection. Mesh processing may include smoothing or decimation.

In one embodiment, a pipelined data flow is implemented through a seriesof data-parallel computations. Real-time, three-dimensionalreconstruction is achieved by organizing each discrete function so thatdata, memory, and GPU cycles are always available. As with otherpipelined systems, the time taken for processing the first series ofcaptured frames will be the total time taken for each function, whereasthe additional time to process successive frames is limited only by thelatency of the slowest discrete function.

As depicted in FIG. 4, when scanning three-dimensional objects, it maybe useful to include a fixture or integrated extension 52 with knownlandmark or fiducial points 54 in order to provide ground truthinformation about the geometry of the three-dimensional scene. Thesefiducial points 54 may be used to assist in aligning data betweenmultiple acquisitions, or in correcting spatial distortions introducedin the Point Cloud construction step (step 44). Thus, in one embodiment,the haptically guided scanning system 10 is provided with an integratedextension 52 that includes fiducial points 54 so that a scanningreference frame may be calculated.

In addition to or instead of including fiducial points 54, theintegrated extension 52 may include special markings geared towardscalibrating the camera 21, such as a three-dimensional range camera.Because the reconstruction algorithms generally assume a known geometryor relationship between the camera 21 and structured-light projector 20,calibration to a known image is important.

Additionally, in some cases, the haptic device 12 may constrain theavailable viewpoints for scanning. For example, the kinematic structureof the haptic device 12, such as the PHANTOM® device, may include stopsor joint limits. To scan the entire object 32, it may therefore benecessary to move, twist, or spin the object 32 during the scanningprocess. Referring again to FIG. 3, an additional twist or othermovement may be corrected in an alignment step (step 48) with theappropriate mathematics. To facilitate this correction, the scanningsystem 10 may include a structure for providing a fixed amount ofrotation (e.g., 90 degrees or 180 degrees) or direct measurement througha rotary position sensor, such as an encoder or potentiometer.

Referring to FIG. 5, in certain embodiments, a haptic device 60 includesa user connection element or pen/stylus 62, a linkage 64, parallellinkages 66 and 68, a linkage disk 70, and a base disk 72. The pen 62 isattached to an end of linkage 64 at a pen joint 74. The pen 62 mayinclude a switch 76, for example, to activate the haptic device 60 orinitiate the scan of an object. An opposite end of the linkage 64 isattached to the two parallel linkages 66, 68 at linkage joints 78, 80.As depicted, the parallel linkages 66, 68 are connected to the linkagedisk 70 at disk joints 82, 84. The linkage disk 70 may be attached tothe base disk 72, which has a central axis that is perpendicular to acentral axis of the linkage disk 70.

The position and orientation of the pen 62 in three-dimensional space iscontrolled with one or more actuators. For example, the haptic device 60may include actuators to rotate the linkage disk 70 and/or the base disk72. In addition, the disk joints 82, 84 may include one or moreactuators to adjust the positions of one or both disk joints 82, 84 onthe linkage disk 70. The linkage 64, parallel linkages 66, 68, linkagedisk 70, base disk 72, and actuators are used to drive the pen 62 up,down, right, left, forwards, and/or backwards, as needed.

In certain embodiments, the pen 62 may be rotated about the pen joint 74in one or more directions. For example, the pen 62 may be rotatablearound a central axis of the pen 62. The pen joint 74 may also include agimbal assembly to allow the pen 62 to be rotated about one or more axesthat are perpendicular to the central axis of the pen 62. The pen joint74 may include one or more actuators to control the orientation of thepen 62.

Referring to FIG. 6, to facilitate the scanning of objects that aredifficult to reach or access, a pen 88 of a haptic device 90 includes anextension joint 92. The extension joint 92 extends from the pen joint 74to a tip 94 where a scanning module 96 is located. As depicted, thescanning module 96 may include a camera and a DLP device. A length ofthe extension joint 92 may be fixed, or it may extendable. For example,the extension joint 92 may extend in the manner of a telescope. Anactuator may be included to extend and/or retract the extension joint92. In certain embodiments, a distance D between the scanning module 96and the pen joint 74 may be between about 1 inch and about 12 inches, orbetween about 2 inches and about 4 inches. In one embodiment, thedistance D is about 3 inches.

Each of the haptic devices described above and depicted in FIGS. 1 and4-6 may be any suitable haptic device. For example, the haptic devicemay be any one of the following haptic devices manufactured by SensAbleTechnologies, Inc. of Wilmington, Mass.: a PHANTOM® Premium 1.5/6DOF or1.5 High force/6DOF haptic device; a PHANTOM® Premium 1.0, 1.5, 1.5 HighForce, or 3.0 haptic device; and a PHANTOM® Premium 3.0/6DOF hapticdevice. The haptic device may include various handles and/or endeffectors.

In certain embodiments, the methods, systems, and apparatus describedherein are configured for use in a Minimally Invasive Surgery (MIS)system. The system may be used to scan anatomical structures, such as aprostate gland, a gall bladder, a pancreas, a stomach, an appendix, aliver, and/or other organs or body parts. In one embodiment, the systemenables a surgeon or other medical professional to view interiorstructures within a patient during evaluation, treatment, or surgery.

Examples of haptic devices that may be used with the system describedherein include those described in the following U.S. patents, thedisclosures of which are all incorporated herein by reference in theirentireties: U.S. Pat. No. 5,898,599, titled, “Force Reflecting HapticInterface,” by Massie et al.; U.S. Pat. No. 6,671,651, titled, “3-DSelection and Manipulation with a Multiple Dimension Haptic Interface,”by Goodwin, et al.; U.S. Pat. No. 6,985,133, titled, “Force ReflectingHaptic Interface,” by Rodomista, et al.; U.S. Pat. No. 7,411,576,titled, “Force Reflecting Haptic Interface,” by Massie, et al.

Examples of modeling systems and user interfaces (e.g., graphical and/orhaptic interfaces) that may be used with the system described hereininclude those described in the following U.S. patents and patentapplications, the texts of which are all incorporated herein byreference in their entireties: pending U.S. patent application Ser. No.12/692,459, titled, “Haptically Enabled Coterminous Production ofProsthetics and Patient Preparations in Medical and DentalApplications,” by Rawley et al., published as U.S. Patent ApplicationPublication No. 2010/0291505; pending U.S. patent application Ser. No.12/321,766, titled, “Haptically Enabled Dental Modeling System,” bySteingart et al., published as U.S. Patent Application Publication No.2009/0248184; pending U.S. patent application Ser. No. 11/998,457,titled, “Systems for Haptic Design of Dental Restorations,” by Steingartet al., published as U.S. Patent Application Publication No.2008/0261165; pending U.S. patent application Ser. No. 11/998,877,titled, “Systems for Hybrid Geometric/Volumetric Representation of 3DObjects,” by Faken et al., published as U.S. Patent ApplicationPublication No. 2008/0246761; U.S. Pat. No. 7,149,596, titled,“Apparatus and Methods for Modifying a Model of an Object to EnforceCompliance with a Manufacturing Constraint,” by Berger et al.; U.S. Pat.No. 7,626,589, titled, “Haptic Graphical User Interface for AdjustingMapped Texture,” by Berger; U.S. Pat. No. 6,958,752, titled, “Systemsand Methods for Three-Dimensional Modeling,” by Jennings, Jr. et al.;U.S. Pat. No. 6,867,770, titled, “Systems and Methods for VoxelWarping,” by Payne; U.S. Pat. No. 6,421,048, titled, “Systems andMethods for Interacting With Virtual Objects in A Haptic Virtual RealityEnvironment,” by Shih et al.; U.S. Pat. No. 6,111,577, titled, “Methodand Apparatus for Determining Forces to be Applied to a User Through aHaptic Interface,” by Zilles et al.; U.S. Pat. No. 7,990,374, titled,“Apparatus and Methods for Haptic Rendering Using Data in a GraphicsPipeline,” by Itkowitz; and pending U.S. patent application Ser. No.11/169,271, titled, “Apparatus and Methods for Haptic Rendering Using aHaptic Camera View,” by Itkowitz, published as U.S. Patent ApplicationPublication No. 2006/0284834.

EQUIVALENTS

While the invention has been particularly shown and described withreference to specific preferred embodiments, it should be understood bythose skilled in the art that various changes in form and detail may bemade therein without departing from the spirit and scope of theinvention as defined by the appended claims. Insofar as this is aprovisional application, what is considered applicants' invention is notnecessarily limited to embodiments that fall within the claims below.

What is claimed is:
 1. A system for haptically enabled,three-dimensional scanning of a tangible object, the system comprising:a haptic interface device configured to provide haptic feedback to auser and receive input from the user during movement of an implement ofthe haptic interface device during three-dimensional scanning of atangible object, wherein the implement comprises a camera; a graphicalinterface configured to provide graphical feedback to the user duringthree-dimensional scanning of the object; and a three-dimensionalscanning application in communication with the haptic interface deviceand the graphical interface, wherein the scanning application, duringscanning of the object: (a) obtains an image of the object captured bythe camera upon activation of a user command; (b) determines, via aprocessor, a haptic guide at one or more positions and/or orientationsin space in relation to the object from which further images areacquired and used for construction of a three-dimensional virtualrepresentation of the object, wherein the haptic guide is determinedusing at least the image obtained in step (a); (c) delivers force to theuser via the haptic interface device according to the haptic guide; (d)repeats one or more of steps (a) to (c) as additional images of theobject are acquired; and (e) produces a three-dimensional virtualrepresentation of the object using a subset of the acquired images,wherein the haptic guide is identified to facilitate further acquisitionof images at the one or more positions and/or orientations in space inrelation to the object to fill voids in the produced three-dimensionalvirtual representation, when used in combination with the previouslyobtained images.
 2. The system of claim 1, wherein the implement of thehaptic interface device comprises a stylus.
 3. The system of claim 1,wherein activation of the user command occurs when the user presses abutton of the haptic interface.
 4. The system of claim 1, wherein theforce is delivered to the user via the haptic interface device toconstrain the movement of the implement by the user to positions and/ororientations corresponding to the haptic guide.
 5. The system of claim1, wherein the system is a part of a minimally invasive surgery (MIS)system.
 6. The system of claim 1, wherein the force is delivered to theuser, via the haptic interface device, to restrict or constrain themovement of the implement to provide a fixed camera focal length.
 7. Amethod for haptically enabled, three-dimensional scanning of a tangibleobject, the method comprising: (a) obtaining an image of the object uponactivation of a user command, wherein the image is obtained, from acamera, during manipulation by the user of an implement of a hapticinterface device about the object, wherein the implement comprises thecamera; (b) delivering graphical feedback to the user via a graphicaldisplay during the manipulation of the implement of the haptic interfacedevice by the user; (c) determining, via a processor, a haptic guide atone or more positions and/or orientations in space in relation to theobject from which further images are acquired during the threedimensional scanning of the tangible object and used for construction ofa three-dimensional virtual representation of the object, wherein thehaptic guide is determined using at least the image obtained in step(a); (d) delivering a force to the user via the haptic interface deviceaccording to the haptic guide; (e) repeating one or more of steps (a) to(d) as additional images of the object are acquired; and (f) producing athree-dimensional virtual representation of the object using a subset ofthe acquired images, wherein the haptic guide is identified tofacilitate further acquisition of images at the one or more positionsand/or orientations in space in relation to the object to fill voids inthe produced three-dimensional virtual representation, when used incombination with the previously obtained images.
 8. The method of claim7, wherein the user command is activated by pressing a button of thehaptic interface device.
 9. The method of claim 7, wherein the implementof the haptic interface device comprises a stylus.
 10. The method ofclaim 7, wherein delivering the force to the user via the hapticinterface device comprises constraining the movement of the implement bythe user to positions and/or orientations corresponding to the hapticguide.
 11. The method of claim 7, wherein the object comprises at leastone member selected from the group consisting of a tooth, a human face,a residual limb, an anatomical structure, and a mechanical part.
 12. Themethod of claim 7, comprising the step of analyzing thethree-dimensional virtual representation to reverse engineer the object.13. The method of claim 7, wherein the force is delivered to the user,via the haptic interface device, to restrict or constrain the movementof the implement to provide a fixed camera focal length.
 14. Anapparatus for scanning a tangible object, comprising: a user connectionelement; a scanning module associated with the user connection element,the scanning module comprising a camera; an actuator; a linkagephysically linking the user connection element to the actuator; and aprocessor for determining force delivered to the user connection elementby the actuator to restrict or guide movement of the user connectionelement according to a haptic guide, wherein the scanning moduleacquires a plurality of images of the object from the camera during thescan to produce a three-dimensional virtual representation, and whereinthe haptic guide restricts or guides the movement of the user connectionelement to provide a fixed camera focal length using a subset of theplurality of images acquired of the object, wherein the haptic guide isidentified to facilitate further acquisition of images at one or morepositions and/or orientations in space in relation to the object to fillvoids in the produced three-dimensional virtual representation, whenused in combination with previously obtained images.
 15. The apparatusof claim 14, wherein the user connection element comprises an extensionjoint for scanning a difficult to reach object.
 16. The apparatus ofclaim 14, wherein the object comprises at least one member selected fromthe group consisting of a tooth, a human face, an anatomical structure,a residual limb, and a mechanical part.
 17. The apparatus of claim 14,wherein the haptic guide facilitates movement of the user connectionelement by the user to one or more positions and/or orientations inspace in relation to the object from which acquisition of images isemployed for construction of a three-dimensional virtual representationof the object from the acquired images.
 18. The apparatus of claim 14,wherein the haptic guide is updated, at least in part, according to oneor more previously obtained images of the object.
 19. The apparatus ofclaim 14, wherein the scanning module is a part of a minimally invasivesurgery (MIS) system.